Models For Speech Recognition Research Articles

Revolutionizing crowd surveillance through voice-driven face recognition empowering rapid identification: towards development of sustainable smart cities

Abstract Recent global efforts to create sustainable smart cities have significantly transformed society and improved the lives of people. Nowadays, crowd surveillance (CS) has become essential in sustainable smart cities and society to protect public safety and security. In this regard, the face-based human detection system has received considerable attention because it is recognized as an emerging method in crowd surveillance applications. Thus, in this work, a new method for real-time identification of people for a crowd surveillance system (CSS) that uses facial and speech recognition technology has been introduced. In traditional CS systems, human operators are frequently used by crowd surveillance systems to watch and evaluate video feeds. Human error and operator weariness may result in lost opportunities or slow replies, which reduce the system's efficacy. Certain procedures, including the initial identification and monitoring of people in video feeds, can be automated using a voice-activated system. To address the issues with the present CSS, a new framework Voice-Activated Face Recognition (VAFR) is proposed in this work. 
The proposed framework combines the speech and face recognition models for crowd surveillance. Experimental and simulation studies have been performed to analyze the performance of the proposed VAFR framework. The proposed framework uses the Viola-Jones algorithm for face identification and the Conformer architecture for speech analysis, reaching a noteworthy 99.7% accuracy rate in live video feeds. In addition, the ethical and safety aspect of the proposed VAFR system is presented.

Multi-view Self-supervised Learning and Multi-scale Feature Fusion for Automatic Speech Recognition

To address the challenges of the poor representation capability and low data utilization rate of end-to-end speech recognition models in deep learning, this study proposes an end-to-end speech recognition model based on multi-scale feature fusion and multi-view self-supervised learning (MM-ASR). It adopts a multi-task learning paradigm for training. The proposed method emphasizes the importance of inter-layer information within shared encoders, aiming to enhance the model’s characterization capability via the multi-scale feature fusion module. Moreover, we apply multi-view self-supervised learning to effectively exploit data information. Our approach is rigorously evaluated on the Aishell-1 dataset and further validated its effectiveness on the English corpus WSJ. The experimental results demonstrate a noteworthy 4.6%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} reduction in character error rate, indicating significantly improved speech recognition performance . These findings showcase the effectiveness and potential of our proposed MM-ASR model for end-to-end speech recognition tasks.

Intelligent speech elderly rehabilitation learning assistance system based on deep learning and sensor networks

Recently, deep learning has been proved to significantly improve the quality of speech recognition. Convolutional neural networks are often used in speech recognition tasks because of their special network structure and powerful learning function. In order to solve the problem that the traditional convolutional neural network can not reflect the one-dimensional basic attributes of speech signal, this paper proposes to set the number of frames for one dimension of convolution kernel, and use one-dimensional model and two-dimensional convolutional network model for speech recognition. By moving the convolution kernel of time axis and frequency band, it can adapt to the time change of speech signal and maintain the correlation between frequency bands to a great extent. At the same time, this paper also discusses the speech signal preprocessing, feature parameter extraction and regularization algorithm. Due to the lack of hospital resources, the lag of information technology, the poor ability of accompanying and other reasons, the current accompanying service for elderly rehabilitation can not meet the needs of elderly patients. The rapid development and wide use of information technology provide opportunities for the optimization of health services for the elderly. In view of the difficulty of word memory and interpretation in current rehabilitation courses, a VR intelligent teaching system based on intelligent voice technology is developed. The application results show that the system can effectively improve the ability of language expression and word writing. At present, the system of intelligent speech function has not been completed, and lacks speech synthesis function. The next research will focus on the use of speech synthesis technology, in order to realize the man-machine dialogue between people and the system, and show a more real training situation.

Deep Learning for Arabic Speech Recognition Using Convolutional Neural Networks

Extracting the speaker's emotional state from their speech has become an active research topic lately due to the demand for more human interactive applications. This field of research has noted significant advancement, especially in the English language, owing to the availability of massive speech-labeled corpora. However, the progress of analogous methodologies in the Arabic language is still in its infancy stages. In this paper, we present a Speech Recognition model for the Arabic language, proficient in discerning both the emotional state and gender of the speaker through voice analysis. Three primary emotion labels were selected: low, standard, and high levels of emotion. Various spectral features, such as the mel-frequency cepstral coefficient (MFCC), were extracted and tested to determine the optimal features. Furthermore, various Machine Learning models (SVM, KNN, and HMM) and Deep Learning models (LSTM and CNN) were evaluated for training. The results were compared between the five models using different extracted features, ultimately culminating in the selection of MFCC, root-mean-square (RMS), mel-scaled spectrogram, spectral, and zero-crossing rate as spectral features, and the CNN as a classification model. This selection yielded significant results, with an accuracy of 93% for emotion recognition and 99% for gender recognition.

Investigation of Imbalanced Sentiment Analysis in Voice Data: A Comparative Study of Machine Learning Algorithms

 INTRODUCTION: Language serves as the primary conduit for human expression, extending its reach into various communication mediums like email and text messaging, where emoticons are frequently employed to convey nuanced emotions. In the digital landscape of long-distance communication, the detection and analysis of emotions assume paramount importance. However, this task is inherently challenging due to the subjectivity inherent in emotions, lacking a universal consensus for quantification or categorization.OBJECTIVES: This research proposes a novel speech recognition model for emotion analysis, leveraging diverse machine learning techniques along with a three-layer feature extraction approach. This research will also through light on the robustness of models on balanced and imbalanced datasets. METHODS: The proposed three-layered feature extractor uses chroma, MFCC, and Mel method, and passes these features to classifiers like K-Nearest Neighbour, Gradient Boosting, Multi-Layer Perceptron, and Random Forest.RESULTS: Among the classifiers in the framework, Multi-Layer Perceptron (MLP) emerges as the top-performing model, showcasing remarkable accuracies of 99.64%, 99.43%, and 99.31% in the Balanced TESS Dataset, Imbalanced TESS (Half) Dataset, and Imbalanced TESS (Quarter) Dataset, respectively. K-Nearest Neighbour (KNN) follows closely as the second-best classifier, surpassing MLP's accuracy only in the Imbalanced TESS (Half) Dataset at 99.52%.CONCLUSION: This research contributes valuable insights into effective emotion recognition through speech, shedding light on the nuances of classification in imbalanced datasets.

Imagined speech classification exploiting EEG power spectrum features.

Imagined speech recognition has developed as a significant topic of research in the field of brain-computer interfaces. This innovative technique has great promise as a communication tool, providing essential help to those with impairments. An imagined speech recognition model is proposed in this paper to identify the ten most frequently used English alphabets (e.g., A, D, E, H, I, N, O, R, S, T) and numerals (e.g., 0 to 9). A novel electroencephalogram (EEG) dataset was created by measuring the brain activity of 30 people while they imagined these alphabets and digits. As part of signal preprocessing, EEG signals are filtered before extracting delta, theta, alpha, and beta band power features. These features are used as input for classification using support vector machines, k-nearest neighbors, and random forest (RF) classifiers. It is identified that the RF classifier outperformed the others in terms of classification accuracy. Classification accuracies of 99.38% and 95.39% were achieved at the coarse-level and fine-level, respectively with the RF classifier. From our study, it is also revealed that the beta frequency band and the frontal lobe of the brain played crucial roles in imagined speech recognition. Furthermore, a comparative analysis against state-of-the-art techniques is conducted to demonstrate the efficacy of our proposed model.

Cluster-Based Pairwise Contrastive Loss for Noise-Robust Speech Recognition.

This paper addresses a joint training approach applied to a pipeline comprising speech enhancement (SE) and automatic speech recognition (ASR) models, where an acoustic tokenizer is included in the pipeline to leverage the linguistic information from the ASR model to the SE model. The acoustic tokenizer takes the outputs of the ASR encoder and provides a pseudo-label through K-means clustering. To transfer the linguistic information, represented by pseudo-labels, from the acoustic tokenizer to the SE model, a cluster-based pairwise contrastive (CBPC) loss function is proposed, which is a self-supervised contrastive loss function, and combined with an information noise contrastive estimation (infoNCE) loss function. This combined loss function prevents the SE model from overfitting to outlier samples and represents the pronunciation variability in samples with the same pseudo-label. The effectiveness of the proposed CBPC loss function is evaluated on a noisy LibriSpeech dataset by measuring both the speech quality scores and the word error rate (WER). The experimental results reveal that the proposed joint training approach using the described CBPC loss function achieves a lower WER than the conventional joint training approaches. In addition, it is demonstrated that the speech quality scores of the SE model trained using the proposed training approach are higher than those of the standalone-SE model and SE models trained using conventional joint training approaches. An ablation study is also conducted to investigate the effects of different combinations of loss functions on the speech quality scores and WER. Here, it is revealed that the proposed CBPC loss function combined with infoNCE contributes to a reduced WER and an increase in most of the speech quality scores.

Artificial Intelligence Powered Voice to Text and Text to Speech Recognition Model – A Powerful Tool for Student Comprehension of Tutor Speech

Speech-to-Text and Text-to-Speech are both NLP(natural language processing) powered models which transform speech to text and vice versa, providing an increased scope of learning for the parties involved. For the past couple of years it's been observed that students have been moving abroad for quality education and better financial aid. Since there is an accent gap between students and tutors which reduces the understanding of students. Our work is done to solve the aforementioned problem. With its state-of-the-art STT(speech-to-text) and TTS(text-to-speech) softwares this work intends to ease the learning curve of the students. The key targets of this work are international students, individuals with disabilities. It can also be used to transcribe meetings for quick conversion of meeting discussion points into text. Companies can also use the model to get the data for the call recordings and further perform sentiment analysis and various such activities. This research aims to give a detailed walk through of the product as it stands, and provide details regarding all aspects of the product. This covers the various tech stacks used, the implementation of the said technologies, the reports shown to the different end users. This provides the workflow of the product.

Customized deep learning based Turkish automatic speech recognition system supported by language model.

In today's world, numerous applications integral to various facets of daily life include automatic speech recognition methods. Thus, the development of a successful automatic speech recognition system can significantly augment the convenience of people's daily routines. While many automatic speech recognition systems have been established for widely spoken languages like English, there has been insufficient progress in developing such systems for less common languages such as Turkish. Moreover, due to its agglutinative structure, designing a speech recognition system for Turkish presents greater challenges compared to other language groups. Therefore, our study focused on proposing deep learning models for automatic speech recognition in Turkish, complemented by the integration of a language model. In our study, deep learning models were formulated by incorporating convolutional neural networks, gated recurrent units, long short-term memories, and transformer layers. The Zemberek library was employed to craft the language model to improve system performance. Furthermore, the Bayesian optimization method was applied to fine-tune the hyper-parameters of the deep learning models. To evaluate the model's performance, standard metrics widely used in automatic speech recognition systems, specifically word error rate and character error rate scores, were employed. Upon reviewing the experimental results, it becomes evident that when optimal hyper-parameters are applied to models developed with various layers, the scores are as follows: Without the use of a language model, the Turkish Microphone Speech Corpus dataset yields scores of 22.2 -word error rate and 14.05-character error rate, while the Turkish Speech Corpus dataset results in scores of 11.5 -word error rate and 4.15 character error rate. Upon incorporating the language model, notable improvements were observed. Specifically, for the Turkish Microphone Speech Corpus dataset, the word error rate score decreased to 9.85, and the character error rate score lowered to 5.35. Similarly, the word error rate score improved to 8.4, and the character error rate score decreased to 2.7 for the Turkish Speech Corpus dataset. These results demonstrate that our model outperforms the studies found in the existing literature.

Summarization of Video using Audio

Abstract: Online education has emerged as a highly effective means of delivering quality education to students. Its popularity has increased due to the high-quality visual and graphical content, delivered by subject matter experts, and the convenience of learning anytime and anywhere. However, students may face time constraints that prevent them from fully engaging with the course content. To address this issue, video transcript summarizers have gained popularity. These tools extract the most important topics from a video, allowing students to understand the essence of the class without having to watch the entire video. Our system focuses on developing a module using txtai.pipeline with Python to summarize online class videos. We use Whisper, a general-purpose speech recognition model, to train our model on a large dataset of diverse audio. The model takes the URL of a video as input and uses two algorithms to summarize the content: TF-IDF and Gensim. The summarization process is subjective, and we have incorporated two prominent methods: cosine similarity and ROUGE score. The former does not require a human-generated summary for reference, while the latter does. Our results show that the efficiency obtained using cosine similarity is greater than 90% in both TF-IDF and Gensim cases. The efficiency obtained in the case of ROUGE score is between 40-50%.

Automating sedation state assessments using natural language processing.

Common goals for procedural sedation are to control pain and ensure the patient is not moving to an extent that is impeding safe progress or completion of the procedure. Clinicians perform regular assessments of the adequacy of procedural sedation in accordance with these goals to inform their decision-making around sedation titration and also for documentation of the care provided. Natural language processing could be applied to real-time transcriptions of audio recordings made during procedures in order to classify sedation states that involve movement and pain, which could then be integrated into clinical documentation systems. The aim of this study was to determine whether natural language processing algorithms will work with sufficient accuracy to detect sedation states during procedural sedation. A prospective observational study was conducted. Audio recordings from consenting participants undergoing elective procedures performed in the interventional radiology suite at a large academic hospital were transcribed using an automated speech recognition model. Sentences of transcribed text were used to train and evaluate several different NLP pipelines for a text classification task. The NLP pipelines we evaluated included a simple Bag-of-Words (BOW) model, an ensemble architecture combining a linear BOW model and a "token-to-vector" (Tok2Vec) component, and a transformer-based architecture using the RoBERTa pre-trained model. A total of 15,936 sentences from transcriptions of 82 procedures was included in the analysis. The RoBERTa model achieved the highest performance among the three models with an area under the ROC curve (AUC-ROC) of 0.97, an F1 score of 0.87, a precision of 0.86, and a recall of 0.89. The Ensemble model showed a similarly high AUC-ROC of 0.96, but lower F1 score of 0.79, precision of 0.83, and recall of 0.77. The BOW approach achieved an AUC-ROC of 0.97 and the F1 score was 0.7, precision was 0.83 and recall was 0.66. The transformer-based architecture using the RoBERTa pre-trained model achieved the best classification performance. Further research is required to confirm the that this natural language processing pipeline can accurately perform text classifications with real-time audio data to allow for automated sedation state assessments. Automating sedation state assessments using natural language processing pipelines would allow for more timely documentation of the care received by sedated patients, and, at the same time, decrease documentation burden for clinicians. Downstream applications can also be generated from the classifications, including for example real-time visualizations of sedation state, which may facilitate improved communication of the adequacy of the sedation between clinicians, who may be performing supervision remotely. Also, accumulation of sedation state assessments from multiple procedures may reveal insights into the efficacy of particular sedative medications or identify procedures where the current approach for sedation and analgesia is not optimal (i.e. a significant amount of time spent in "pain" or "movement" sedation states).

SPEECH RECOGNITION SYSTEM

Speech recognition technology has witnessed remarkable progress in recent years, fueled by advancements in machine learning, deep neural networks, and signal processing techniques. This paper presents a comprehensive review of the current state-of-the-art in speech recognition systems, highlighting key methodologies and breakthroughs that have contributed to their improved performance. The paper explores various aspects, including acoustic modeling, language modeling, and the integration of contextual information, shedding light on the challenges faced and innovative solutions proposed in the field. Furthermore, the paper discusses the impact of large-scale datasets and transfer learning on the robustness and adaptability of speech recognition models. It delves into recent developments in end-to-end models and their potential to simplify the architecture while enhancing accuracy. The integration of real-time and edge computing for speech recognition applications is also explored, emphasizing the implications for practical implementations in diverse domains such as healthcare, telecommunications, and smart devices. In addition to reviewing the current landscape, the paper provides insights into future prospects and emerging trends in speech recognition research. The role of multimodal approaches, incorporating visual and contextual cues, is discussed as a potential avenue for further improvement. Ethical considerations related to privacy and bias in speech recognition systems are also addressed, emphasizing the importance of responsible development and deployment. By synthesizing current research findings and anticipating future directions, this paper contributes to the evolving discourse on speech recognition technologies, providing a valuable resource for researchers, practitioners, and industry professionals in the field. Key Words: Real-time processing , Machine learning , Deep neural networks , Technology advancements , Contextual information , Large-scale datasets Transfer learning , End-to-end models , Real-time processing Edge computing , Multimodal approaches Ethical considerations , Privacy , Bias , Future prospects Research review.

Speech recognition model design for Sundanese language using WAV2VEC 2.0

Speech recognition model design for Sundanese language using WAV2VEC 2.0

Research on the Construction of English Translation Model for Speech Recognition Based on Multiple Information Sources and Lexical Assistance

Research on the Construction of English Translation Model for Speech Recognition Based on Multiple Information Sources and Lexical Assistance

Assessment and analysis of accents in air traffic control speech: a fusion of deep learning and information theory.

Enhancing the generalization and reliability of speech recognition models in the field of air traffic control (ATC) is a challenging task. This is due to the limited storage, difficulty in acquisition, and high labeling costs of ATC speech data, which may result in data sample bias and class imbalance, leading to uncertainty and inaccuracy in speech recognition results. This study investigates a method for assessing the quality of ATC speech based on accents. Different combinations of data quality categories are selected according to the requirements of different model application scenarios to address the aforementioned issues effectively. The impact of accents on the performance of speech recognition models is analyzed, and a fusion feature phoneme recognition model based on prior text information is constructed to identify phonemes of speech uttered by speakers. This model includes an audio encoding module, a prior text encoding module, a feature fusion module, and fully connected layers. The model takes speech and its corresponding prior text as input and outputs a predicted phoneme sequence of the speech. The model recognizes accented speech as phonemes that do not match the transcribed phoneme sequence of the actual speech text and quantitatively evaluates the accents in ATC communication by calculating the differences between the recognized phoneme sequence and the transcribed phoneme sequence of the actual speech text. Additionally, different levels of accents are input into different types of speech recognition models to analyze and compare the recognition accuracy of the models. Experimental results show that, under the same experimental conditions, the highest impact of different levels of accents on speech recognition accuracy in ATC communication is 26.37%. This further demonstrates that accents affect the accuracy of speech recognition models in ATC communication and can be considered as one of the metrics for evaluating the quality of ATC speech.

Model Access Control Based on Hidden Adversarial Examples for Automatic Speech Recognition

DNNs have achieved remarkable success across various domains, and their commercial value has led to their classification as intellectual property (IP) for their creators. While model watermarking is commonly employed for DNN IP protection, it is limited to post-hoc forensics. In contrast, model access control offers a more effective, proactive approach to prevent IP infringement through authentication. However, existing model access control methods primarily focus on image classification models and are not suitable for automatic speech recognition (ASR) models, which are also widely used in commercial applications. To address the above limitation, inspired by audio adversarial examples, we propose the first model access control scheme for IP protection of ASR models, which utilizes audio adversarial examples with target labels as user identity information, serving as identity-proof samples. However, a unique challenge arises in the form of interception attacks, in which an attacker detects and hijacks an authorized sample to bypass the authentication process. To remedy it, we introduce the hidden adversarial examples (HAEs) for authentication, which embed the authorized information by slightly modifying the logits and behaving like clean audios, thereby making them difficult to be detected by analyzing the predicted results. To further evade detection by steganalysis, which can be employed for adversarial example detection, we design a distortion cost function inspired by adaptive steganography to guide the generation of HAEs. We conduct extensive experiments on the open-source ASR system DeepSpeech, demonstrating that our proposed scheme effectively protects ASR models proactively and is resistant to unique interception attacks.

ANALYSIS OF WHISPER AUTOMATIC SPEECH RECOGNITION PERFORMANCE ON LOW RESOURCE LANGUAGE

Implementing Automatic Speech Recognition Technology in daily life could give convenience to its users. However, speeches that can be recognized accurately by the ASR model right now are in languages considered high resources, like English. In previous research, a few regional languages like Javanese, Sundanese, Balinese and Btaknese are used in automatic speech recognition. This research aim is to improve speech recognition using the ASR model on low-resource language. The dataset used in this research is the Javanese dataset specifically because there is a high-quality Javanese speech dataset provided by previous research. The method used is fine-tuning the Whisper model which has been trained on 680,000 hours of multilingual voice data using a Javanese speech dataset. To reduce computation requirements, parameter efficient fine-tuning (PEFT) implemented in the fine-tuning process. The trainable parameter is reduced to <1% because the implementation of PEFT reduces the computation required by the model for fine-tuning. The best WER evaluation result is 13.77%, achieved by the fine-tuned Whisper large-v2 model compared to the base model of Whisper large-v2, which achieves 89.40% in WER evaluation. Performance improvement in WER evaluation showed that fine-tuning effectively improves the performance of the Whisper automatic speech recognition model on recognizing speeches in low-resource languages like the Javanese language compared to the Original Whisper model performance with minimal computational cost needed for fine-tuning large model.

Speech Recognition for Smart AI Devices

This article looks at the progress, challenges and future directions in the field of speech recognition for intelligent AI devices. Speech recognition technology has made significant progress over the years, enabling seamless interactions between humans and artificial intelligence. This article provides an in-depth analysis of modern speech recognition techniques used in smart AI devices and the potential applications and limitations of the technology. It also discusses the current challenges faced by developers and researchers and suggests possible solutions to overcome them. Keywords-Natural language processing Neat vs. scruff, Soft vs. hard computing, Narrow vs general AI, Rule-based approaches, Statistical and machine learning based methods, Deep Neural Networks, Deep learning models for speech recognition, Hybrid and ensemble approaches, Voice biometrics and authentication, Speech translation and language learning, Natural language processing improvement.

Joint streaming model for backchannel prediction and automatic speech recognition

AbstractIn human conversations, listeners often utilize brief backchannels such as “uh‐huh” or “yeah.” Timely backchannels are crucial to understanding and increasing trust among conversational partners. In human–machine conversation systems, users can engage in natural conversations when a conversational agent generates backchannels like a human listener. We propose a method that simultaneously predicts backchannels and recognizes speech in real time. We use a streaming transformer and adopt multitask learning for concurrent backchannel prediction and speech recognition. The experimental results demonstrate the superior performance of our method compared with previous works while maintaining a similar single‐task speech recognition performance. Owing to the extremely imbalanced training data distribution, the single‐task backchannel prediction model fails to predict any of the backchannel categories, and the proposed multitask approach substantially enhances the backchannel prediction performance. Notably, in the streaming prediction scenario, the performance of backchannel prediction improves by up to 18.7% compared with existing methods.

Multimodal audiovisual speech recognition architecture using a three‐feature multi‐fusion method for noise‐robust systems

AbstractExposure to varied noisy environments impairs the recognition performance of artificial intelligence‐based speech recognition technologies. Degraded‐performance services can be utilized as limited systems that assure good performance in certain environments, but impair the general quality of speech recognition services. This study introduces an audiovisual speech recognition (AVSR) model robust to various noise settings, mimicking human dialogue recognition elements. The model converts word embeddings and log‐Mel spectrograms into feature vectors for audio recognition. A dense spatial–temporal convolutional neural network model extracts features from log‐Mel spectrograms, transformed for visual‐based recognition. This approach exhibits improved aural and visual recognition capabilities. We assess the signal‐to‐noise ratio in nine synthesized noise environments, with the proposed model exhibiting lower average error rates. The error rate for the AVSR model using a three‐feature multi‐fusion method is 1.711%, compared to the general 3.939% rate. This model is applicable in noise‐affected environments owing to its enhanced stability and recognition rate.